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ABSTRACT 

The  advancing  state  of  the  art  in  dismounted  embedded  training  makes  use  of  helmet- mounted  displays,  man- 
wearable  computers,  and  other  immersive  hardware  to  construct  increasingly  engaging  environments.  Within  such  a 
framework,  structured  training  methods  provide  a  means  to  achieve  learning  objectives  and  concept  retention,  with 
minimal  instructor  involvement.  Intelligent  stmctured  training  applies  real-time  automated  evaluation  and  feedback 
methods  based  on  Intelligent  Tutoring  Systems  (ITS)  techniques.  This  paper  reviews  results  from  the  integration  of 
an  Intelligent  Stmctured  Trainer  with  the  embedded  Virtual  Warrior  Soldier  prototype  developed  for  the  Army 
RDECOM  Simulation  and  Training  Technology  Center.  Army  subject  matter  experts  defined  dismounted  training 
objectives  and  specific  requirements  for  integrated  evaluation  mechanisms.  The  paper  discusses  three  areas  of 
research  results,  both  in  terms  of  direct  research  findings  and  also  how  these  findings  can  be  applied  for  future  work. 
First,  the  effort  identified  the  nature  of  the  data  that  an  integrated  structured  trainer  consumes  in  order  to  generate 
useful  real-time  feedback  for  dismounted  Soldiers.  This  data  includes  not  only  state  information  direct  from  the 
simulation,  but  also  data  reflecting  Soldier  actions  in  the  primary  interface  and  secondary  Command  and  Control 
interfaces.  Data  categories  can  be  generalized  and  catalogued  for  future  related  training  efforts.  The  second 
research  outcome  is  an  analysis  of  scenario  authoring  requirements,  in  terms  of  SAF  (semi-automated  forces) 
performance,  terrain  database  accuracy  and  consistency,  data  protocols  and  availability,  and  the  authoring  process 
itself.  Third,  user  feedback  collected  from  initial  experiments  with  human  Participants  provide  several  indicators  for 
the  areas  of  greatest  fit  or  friction  between  the  dismounted  training  objectives  and  a  structured  training  approach. 
This  helps  define  the  road  ahead,  in  setting  goals  for  the  usability  and  realism  of  the  training  environment,  in 
identifying  the  dismounted  task  areas  best  suited  to  structured  training,  and  in  applying  varied  methods  for 
automated  feedback  delivery . 
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INTRODUCTION 

The  advancing  state  of  the  art  in  dismounted  embedded 
training  makes  use  of  helmet- mounted  displays,  man- 
wearable  computers,  and  other  immersive  hardware  to 
construct  increasingly  engaging  environments.  Within 
such  a  framework,  structured  training  methods  provide 
a  means  to  achieve  learning  objectives  and  concept 
retention,  with  minimal  instructor  involvement. 
Intelligent  structured  training  applies  real-time 
automated  evaluation  and  feedback  methods  based  on 
Intelligent  Tutoring  Systems  (ITS)  techniques.  This 
paper  reviews  findings  from  the  integration  of  an 
Intelligent  Structured  Trainer  with  the  embedded 
Virtual  Warrior  Soldier  prototype  developed  for  the 
Army  Research  Development  and  Engineering 
Command  (RDECOM)  Simulation  and  Training 
Technology  Center  (STTC).  Army  subject  matter 
experts  defined  dismounted  training  objectives  and 
specific  requirements  for  integrated  evaluation 
mechanisms.  The  paper  discusses  three  areas  of 
research  results,  both  in  terms  of  direct  research 
findings  and  also  how  these  findings  can  be  applied  for 
future  work. 

First,  the  effort  identified  the  nature  of  the  data  that  an 
integrated  structured  trainer  consumes  in  order  to 
generate  useful  real-time  feedback  for  dismounted 
Soldiers.  These  data  include  not  only  state  information 
direct  from  the  simulation,  but  also  data  reflecting 
Soldier  actions  in  the  primary  interface  and  secondary 
Command  and  Control  interfaces.  Data  categories  can 
be  generalized  and  catalogued  for  future  related 
training  efforts. 

The  second  research  outcome  is  an  analysis  of  scenario 
authoring  requirements  following  the  process  of 
constructing  scenarios  for  the  structured  trainer  and 
also  developing  a  preliminary  scenario  authoring  tool. 


This  led  to  new  insights  regarding  requisite 
performance  criteria  in  terms  of  modeled  behaviors  in 
semi -automated  forces  (SAF),  terrain  database 
accuracy  and  consistency,  data  protocols  and 
availability,  and  the  authoring  process  itself. 

Third,  user  feedback  collected  from  initial  experiments 
with  human  Participants  provide  several  indicators  for 
the  areas  of  greatest  fit  or  friction  between  the 
dismounted  training  objectives  and  a  structured 
training  approach.  This  helps  define  the  road  ahead,  in 
setting  goals  for  the  usability  and  realism  of  the 
training  environment,  in  identifying  the  dismounted 
task  areas  best  suited  to  structured  training,  and  in 
applying  varied  methods  for  feedback  delivery. 


BACKGROUND 

Previous  Structured  Training  Experiments 

This  research  builds  on  a  previous  effort  which 
integrated  an  Intelligent  Structured  Trainer  with  a 
Command  and  Control  Vehicle  (C2V)  crewstation 
testbed  for  training  experiments  in  a  task  area  under  the 
Army’s  Future  Combat  Systems  (FCS)  concept. 
Within  this  task  area,  the  FCS  Soldier  manning  a 
Command  and  Control  Vehicle  (C2V)  crewstation 
testbed  must  remotely  control  robotic  platforms  to 
perform  reconnaissance  and  engage  the  enemy.  The 
objective  was  to  mirror  the  embedded  training  context, 
where  a  great  deal  of  future  force  training  is  planned  to 
occur,  and  where  instructors  will  not  be  available  to 
provide  tailored  feedback  during  the  execution  of  an 
exercise. 

In  a  bi-directional  integration  with  the  C2V  testbed, 
evaluation  mechanisms  monitor  events  in  the 
simulation  and  testbed  in  real  time,  and  deliver 
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feedback  during  scenario  execution  as  appropriate  via 
messages  published  to  the  simulation  environment. 
This  procedure  is  performed  by  intelligent  agents 
indexed  to  evaluation  principles  identified  by  subject 
matter  experts.  The  agents  are  implemented  in 
Behavior  Transition  Networks  (BTNs),  which  resemble 
finite  state  machines  with  several  extensions  to  make 
them  more  powerful  for  applications  like  hierarchical 
principle -based  evaluations.  The  system  also  makes 
use  of  a  toolset  for  BTN  authoring  and  execution  (Fu  et 
al,  2003).  The  structured  training  experiments  yielded 
significant  findings  in  three  result  categories: 

•  Retention  of  procedural  knowledge 

•  Quantity  of  procedural  errors 

•  Retention  of  conceptual  knowledge 

For  each  of  these  categories,  results  were  compared 
between  two  cases.  The  first  case  was  automated 
immediate  feedback  from  the  structured  trainer  during 
the  exercise.  The  second  case  was  human-led  after 
action  review,  where  the  only  feedback  was  delayed 
until  post-exercise.  Immediate  directive  feedback  was 
found  to  have  a  significant  effect  in  reducing  the 
number  of  errors  committed  while  acquiring  new 
procedural  skills  during  training,  as  well  as  retention  of 
these  procedural  skills.  Specifically,  the  immediate 
feedback  case  resulted  in  significantly  fewer 
procedural  errors  than  the  delayed  feedback  case.  The 
two  cases  had  comparable  performance  in  retention  of 
procedural  knowledge,  and  delayed  feedback  had  better 
results  in  retention  of  conceptual  knowledge.  However 
in  each  result  category,  performance  measurements 
supported  the  conclusion  that  learning  results  did 
successfully  occur  from  the  use  of  the  structured  trainer 
(Jensen  et  al.  2005). 

These  experiments  provided  a  basis  for  the  effort, 
reported  here,  which  aimed  to  port  the  same  structured 
training  approach  to  a  markedly  different  application, 
training  small-unit  dismounted  Infantry  operations.  By 
doing  so,  the  goal  was  to  highlight  key  training  factors 
unique  to  the  dismounted  platform  and  lay  out  a 
roadmap  for  the  accomplishment  of  learning  results 
similar  to  those  achieved  with  the  C2V  testbed  and 
trainer. 

Virtual  Dismounted  Training 

Use  of  virtual  reality  systems  for  the  individual 
Infantry  Soldier  has  progressed  quickly  over  the  last 
ten  years.  The  U.S.  Army  is  fielding  specific 
simulators  that  allow  the  Infantryman  to  train  in  virtual 
reality.  Prior  to  the  fielding  of  these  systems,  the 
Infantryman  was  largely  ignored  mainly  due  to  the  lack 
of  available  technology.  These  systems  are  housed  in 


dedicated  simulation  centers  which  units  visit  to  train 
in  their  mission  essential  tasks.  This  system  is  named 
the  Squad  Synthetic  Environment  (SSE)  (Chisholm, 
2003). 

This  virtual  hfantry  training  system  is  located  in  the 
Dismounted  Battlespace  Battle  Lab  (DBBL)  at  Fort 
Benning,  Georgia.  The  DBBL  is  located  in  a 
warehouse  type  building  and  is  one  of  the  first  systems 
attempting  to  provide  a  fully  immersive  environment 
for  the  Infantry  Soldier  (Rodriguez,  2003) . 

The  Squad  Synthetic  Environment  is  based  on  the 
Soldier  Visualization  System  (SVS),  manufactured  by 
Advanced  Interactive  Systems  (AIS).  The  Squad 
Synthetic  Environment  (SSE)  is  used  in  two  forms:  a 
fully  immersive  system  that  uses  a  CAVE,  and  a 
desktop  version.  The  CAVE  systems  are 
approximately  15  feet  by  30  feet  by  10  feet  in 
dimension  (Miller,  2002),  with  sensors  to  determine  the 
Soldier’s  location  and  posture.  The  helmet  and  weapon 
also  have  sensors  to  help  determine  the  Soldier’s 
viewpoint,  and  the  location  of  any  simulated  rounds 
that  he  may  fire  (Gately,  2005).  The  other  form  is 
simply  a  desktop  version,  which  has  the  same 
capabilities  as  the  CAVE  systems  in  terms  of  weapon 
selection  and  communications  (Miller,  2002).  Using 
the  desktop  version  is  an  experience  similar  to  many 
first  person  shooter  games  marketed  to  gamers. 

The  SSE  has  allowed  the  U.S  Army  to  explore  virtual 
training  for  the  dismounted  Infantry  Soldier.  A  critical 
operational  limitation  of  this  system  is  its  availability. 
Fort  Benning,  Georgia  b  responsible  for  training  all 
Infantry  Soldiers  from  the  ranks  of  Private  to  Captain. 
The  post  also  is  the  home  of  three  Infantry  battalions 
and  a  Ranger  battalion.  There  are  over  three  thousand 
Soldiers  available  for  training  each  day,  while  the 
DBBL  can  only  provide  eighty  stand-up  modules  for 
training  (Gately,  2005).  This  is  most  likely  not  enough 
systems  to  provide  a  significant  training  benefit  to  the 
commanders  and  trainers  who  are  responsible  for 
training  Soldiers.  Desktop  trainers  can  supplement  the 
CAVE  systems.  The  British  Army  has  conducted  a 
study  on  the  effectiveness  of  desktop  trainers  for  urban 
operations.  Naturally,  most  Soldiers  surveyed 
indicated  that  they  prefer  to  train  in  a  live  environment. 
Most  Soldiers  also  felt  that  the  desktop  trainer  would 
be  a  useful  tool  for  the  leaders  to  sharpen  their  decision 
making  and  command  and  control  skills.  Slight 
performance  improvement  has  also  been  reported  in  a 
section  that  received  only  virtual  training  before 
performing  the  collective  task  in  the  live  environment. 
No  instances  of  negative  training  were  noted  from 
those  surveyed  who  trained  in  the  virtual  environment 
(Pennel,  2003). 
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Dismounted  Player  Unit  Testbed 

The  testbed  for  the  research  reported  in  this  paper  was 
a  prototype  embedded  trainer  developed  by  General 
Dynamics,  named  Virtual  Warrior  (VW).  The  Virtual 
Warrior  system  (shown  in  Figure  1)  incorporates 
movement  tracking  sensors  with  six  degrees  of 
freedom,  independently  tracking  the  head,  leg,  torso, 
and  weapon.  These  sensors  are  integrated  into  the 
hardware  and  software  interfaces. 


Figure  1.  Virtual  Warrior  Prototype 


There  are  two  physical  controllers  used  with  the  VW; 
the  Weapon  User  Interface  or  WUI,  and  the  Soldier 
Control  Unit  Interface  or  SCUI.  The  WUI  is 
essentially  a  joystick  that  allows  the  user  to  move 
forward,  backwards,  left,  right,  and  toggle  through 
certain  tasks  such  as  bore  sighting  and  marking  cleared 
rooms.  The  SCUI  allows  the  user  to  view  the 
situational  awareness  screen,  send  digital  messages, 
use  mouse  controls,  change  posture,  and  reset  the 
system. 

Soldiers  experience  the  virtual  environment  visually 
with  the  use  of  a  helmet  mounted  display  (HMD) 
which  uses  the  existing  bracket  used  for  night  vision 
goggles.  The  HMD  is  manufactured  by  eMagin,  and 
provides  an  800x600  binocular  resolution  and  a  28 
degree  diagonal  field  of  view.  Stereo  sound  is  also 
provided  through  speakers  that  have  been  adapted  to 
the  helmet.  A  specially  implemented  message  box  was 
developed  as  a  popup  that  can  appear  in  the  3D  view 
when  the  integrated  Intelligent  Structured  Trainer 
detected  conditions  meriting  immediate  feedback. 
Figure  2  shows  a  typical  view  with  the  popup  window 
present. 


_ Viitual  Warrioi  U  |[p]^ 


Figure  2.  Testbed  3D  View  with  Feedback  Popup 

The  Virtual  Warrior  uses  the  APEX  advanced  game 
rendering  engine.  This  engine  is  interoperable  with  DIS 
protocols,  HLA  protocols,  and  Open  Flight  database. 
The  underlying  simulation  used  for  the  scenarios 
employed  with  these  experiments  was  the  OneSAF 
Testbed  Baseline  (OTB).  Although  the  architecture 
could  support  multiple  players,  the  scenario  was 
designed  to  operate  with  one  human  Soldier  and  SAF 
behaviors  for  the  other  team  members. 

The  design  of  the  experimental  scenarios  with  the  VW 
player  unit  also  incorporated  the  use  of  the  C2  Mobile 
Intelligent  Net-centric  Computing  System  (C2MINCS) 
into  the  testbed  architecture.  This  is  a  dismounted 
mobile  computing  platform  to  provide  Soldiers 
network  centric  C4ISR  connectivity  and  situational 
awareness.  Specifically  for  these  scenarios  C2MINCS 
was  used  to  send  a  variety  of  pre -formatted  tactical 
reports,  including  spot  reports,  position  reports,  and 
target  handoff  reports . 


DATA  COLLECTION  REQUIREMENTS 

A  common  thread  in  training  applications  with 
automated  evaluation  goals ,  and  particularly  in  ITS 
systems,  is  the  need  to  capture  as  much  data  as  possible 
to  correctly  interpret  the  decision-making  of  the 
trainees.  Frequently  the  cognitive  skills  required  for 
military  operations  fall  into  two  categories:  conceptual 
knowledge  and  procedural  knowledge.  Conceptual 
knowledge  includes  general  tactical  principles  and 
underlying  purposes  which  provide  a  framework  for 
applying  procedural  knowledge.  For  example, 
conceptual  knowledge  includes  understanding  the 
tactical  principles  of  clearing  a  building,  and  the 
purposes  for  tasks  like  submitting  reports.  Procedural 
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knowledge  is  the  understanding  of  the  specific 
sequence  of  subtasks,  such  as  the  proper  position 
within  a  formation  in  movement.  The  major  categories 
of  data  collection  requirements  for  the  implementation 
of  automated  evaluations  for  these  forms  of  knowledge 
in  this  domain  included: 

•  Simulation  entity  state  data  (e.g.,  position, 
orientation,  posture,  head  orientation) 

•  Operator  inputs  (e.g.,  reports  sent,  vehicle 
mount/dismount  actions,  use  of  controls) 

•  Line-of-sight  data  (e.g.,  calculations  for  pairs 
of  positions) 

•  Terrain  feature  data  (e.g.,  coordinates  that  lie 
inside  buildings) 

This  data  is  consumed  in  evaluation  mechanisms 
defined  in  a  Behavior  Transition  Network  (BTN) 
format.  The  notion  of  underlying  forms  of  knowledge 
or  skills  has  a  counterpart  in  a  hierarchy  of  principles 
defined  for  the  domain  by  the  scenario  author.  In  run¬ 
time,  individual  BTNs  may  perform  evaluation 
functions  for  either  single  or  multiple  principles, 
depending  on  whether  efficiencies  of  scale  can  be 
realized  in  terms  of  data  analysis.  For  example,  in  a 
case  where  two  tactical  principles  would  be  evaluated 
via  analysis  of  the  same  set  of  source  data,  then  both 
can  be  implemented  together  in  the  same  BTN,  even  if 
it  produces  different  conclusions  with  respect  to  the 
different  principles  in  different  execution  states. 

For  a  simple  example  of  the  data  required  by  a  specific 
evaluation  principle,  consider  the  task  of  evaluating  for 
a  designated  team  member  maintaining  proper  position 
within  a  fire  team  formation  during  movement.  Figure 
3  below  shows  a  proper  wedge  formation  for  a  four 
person  fire  team. 

e  * 

Direction 
of  movement 


o 

Figure  3.  4-person  Fireteam  Wedge  Formation 

For  illustration,  this  example  describes  a  sinplified 
tiered  logic  that  evaluates  team  member  R  for  proper 
execution  during  free-play  execution.  In  the  first  tier, 
the  evaluation  must  know  when  it  applies,  or  more 
specifically,  when  the  fire  team  intends  to  be  moving  in 


wedge  formation.  There  are  many  ways  to  establish 
this,  but  a  simple  method  is  to  simply  annotate  the 
terrain.  The  second  tier  of  the  evaluation  involves  a 
simple  check  for  relative  distance  between  R  and  G, 
without  any  concern  for  the  positions  of  other  team 
members,  as  shown  in  Figure  4. 


Figure  4.  Distance  Evaluation 

If  R  is  greater  than  10m  from  G,  in  any  direction,  then 
error  conditions  have  been  met,  and  the  evaluation 
triggers  a  feedback.  In  Figure  4,  R  is  out  of  position, 
being  too  far  from  G.  However,  if  this  test  is  passed, 
then  the  third  tier  of  the  logic  is  queued.  On  the 
assumption  that  R  is  within  the  proper  distance  radius 
of  G,  the  next  step  is  to  check  if  R  is  in  the  proper 
section  of  that  radius  (area  A  in  Figure  5). 


Figure  5.  Distance  and  Position  Evaluation 

Area  A  is  defined  by  taking  the  heading  from  G  to  TL, 
and  calculating  the  space  of  a  pie  wedge  created  by  an 
offset  on  either  side  of  that  heading.  In  the  figure 
above,  R  is  in  a  position  that  would  satisfy  the  second 
tier  test  (position  within  the  10m  radius),  but  fail  the 
third  tier  test  (position  outside  of  area  A). 

This  example  contains  several  simplifications  from 
real-world  tactical  concepts,  and  also  depends  on  a 
specific  artifact  of  how  this  evaluation  would  be  used 
in  training  -  namely  that  the  position  of  G  relative  to 
the  team  leader  TL  will  always  be  correct  because  all 
other  entities  are  controlled  by  SAF.  However,  it  is 
useful  for  illustrating  two  things.  First,  data 
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requirements  can  be  simplified  with  respect  to 
operational  definitions  based  on  tactics,  techniques  and 
procedures,  when  the  range  of  use  cases  establishes 
constraints  that  reduce  the  space  of  possible 
interpretations.  In  the  above  example,  it’s  not 
necessary  to  get  an  explicit  datum  for  the  direction  of 
movement,  because  this  is  derived  from  the  SAF 
positions  and  movement.  Second,  the  tiered 
implementation  is  useful  not  only  for  simplifying 
processing  (i.e.,  when  the  second  tier  test  is  failed,  it’s 
not  necessary  to  perform  the  third  tier  test),  but  also  for 
identifying  nuances  of  the  conditions  in  which 
feedback  conditions  occurred.  Although  the  same 
operational  principle  is  violated  with  a  failure  of  either 
the  second  or  third  tier  tests,  by  distinguishing  the 
conditions  in  the  BTN  implementation  it  becomes 
possible  to  generate  different  (and  therefore  more 
informative)  feedback  content  in  the  two  cases. 
Finally,  once  the  BTN  for  such  a  principle  is 
implemented,  it  can  be  easily  parameterized  and 
abstracted  for  rapid  instantiation  in  any  scenario 
requiring  the  same  evaluation  capability,  assuming  the 
same  constraints  apply. 


SCENARIO  AUTHORING  REQUIREMENTS 

One  of  the  focuses  of  this  research  was  to  define  the 
requirements  and  capabilities  for  authoring  tools  that 
would  allow  a  subject  matter  expert  or  other  scenario 
author  to  construct  the  training  situations  where  the 
desired  skills  and  tasks  will  be  exercised.  Scenario 
development  involves  the  choice  of  locations, 
placement  of  entities  and  routes,  and  definition  of 
opposing  force  behaviors  to  make  the  scenario  “play” 
as  intended  for  a  given  set  of  training  objectives.  This 
can  be  a  difficult  process  if  available  tools  require  the 
scenario  to  be  defined  in  a  2D  environment  such  as  the 
OTB  simulation  itself,  while  the  trainee  experience  is 
largely  in  the  3D  visualization  environment.  As  a 
result,  the  scenario  author  must  be  familiar  with  both 
tools,  and  must  be  able  to  cross-reference  any  desired 
locations  in  the  3D  world  with  the  2D  world  where 
they  will  be  saved.  In  addition,  the  scenario  author 
must  also  think  about  which  evaluations  should  apply, 
along  with  any  input  parameters  they  require,  and  this 
must  again  be  cross-referenced  in  both  worlds. 


Warrior  player  unit.  Figure  6  shows  the  main  view  of 
the  authoring  tool. 


Figure  6.  Dismount  View  from  Point  of  Interest 

In  this  view,  a  point  of  interest  has  been  placed  in  the 
3D  environment,  and  a  dismounted  entity  has  been 
placed  as  well,  acting  as  a  form  of  avatar.  Navigating 
in  the  environment,  the  author  can  see  both  the  location 
of  the  dismounted  entity  from  an  offset  camera 
position,  and  the  view  from  the  perspective  of  the 
entity.  This  is  helpful  for  constructing  contact  points  in 
the  scenario.  A  second  view  is  shown  in  Figure  7,  to 
illustrate  the  process  of  applying  an  evaluation  to  the 
scenario. 


Therefore,  in  addition  to  the  implementation  of  the 
training  architecture  integrated  with  the  VW  testbed,  a 
preliminary  prototype  authoring  tool  was  developed  as 
a  platform  from  which  to  identify  the  technical 
challenges  and  functional  requirements  for  a  usable 
toolset.  This  prototype  was  built  on  top  of  the  3D 
Mission  Planner  environment  provided  by  General 
Dynamics,  for  its  compatibility  with  the  Virtual 


Figure  7.  Applying  an  Evaluation  to  the  Scenario 

Since  an  Intelligent  Structured  Trainer  uses  automated 
evaluations  based  on  ITS  technology,  flexible 
application  to  different  scenarios  relies  on  the 
flexibility  of  available  evaluation  mechanisms .  For  the 
evaluations  constructed  with  this  prototype,  their 
representations  were  abstracted  such  that  the  process  of 
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instantiating  them  for  a  specific  scenario  is  a  matter  of 
filling  in  parameters,  as  opposed  to  constructing  similar 
logic  for  each  scenario.  Typical  parameters  include 
specific  coordinates  or  areas  in  the  virtual  space, 
named  entities  or  objects,  routes  or  points  of  interest, 
and  specific  values  governing  the  application  of  the 
evaluation  itself.  Figure  7  shows  the  instantiation  of  an 
evaluation  for  the  principle  of  reporting  position  upon 
arrival  at  a  checkpoint.  The  location  parameters  are 
automatically  taken  from  the  point  placed  in  3D,  and 
additional  parameters  for  the  evaluation  itself  establish 
the  delay  before  feedback  should  be  produced.  The 
evaluation  dialog  also  allows  the  author  to  edit  the 
feedback  text  that  will  be  generated. 

Generalizing  from  this  example,  the  design  of  the 
training  approach  necessitates  a  model  where 
automated  evaluations  are  constructed  in  a  manner 
such  that  they  can  automatically  adapt  themselves  to  a 
range  of  scenarios  for  the  domain,  including  those  not 
yet  defined.  A  framework  is  necessary  so  that  the 
scenario  authoring  process  can  refer  directly  to 
automated  evaluations  defined  for  the  domain.  This  is 
largely  accomplished  with  the  organization  of 
evaluation  behaviors  in  a  modular  structure  as 
individual  BTNs  that  can  be  simply  instantiated  with  a 
given  scenario. 

Through  the  process  of  developing  the  initial  prototype 
authoring  tool  and  reviewing  how  it  could  be  used  for 
different  kinds  of  scenarios  and  sets  of  evaluations,  the 
following  observations  were  collected: 

•  It’s  important  to  use  a  visualization  tool  as  the 
backbone  for  the  authoring  tool,  so  that  the 
scenario  author  can  easily  fly  through  terrain 
to  pick  locations  directly  in  the  3D  view,  and 
have  the  locations  automatically  collected  and 
saved  in  a  scenario  file.  This  affords  the 
scenario  author  with  the  ability  to  see 
positions  and  sightlines  as  the  training 
audience  will  see  them  as  they  encounter 
routes,  terrain  objects,  and  vehicles  in  the 
scenario  execution.  The  result  is  a  direct 
procedure  for  defining  the  “gameplay”  of  a 
scenario  without  requiring  trial  and  error 
toggling  between  the  2D  and  3D  components 
of  the  testbed. 

•  The  ideal  authoring  tool  use  case  does  not 
require  connectivity  to  the  specific  simulation 
component  of  the  testbed  architecture.  This 
reduces  the  installed  profile  necessary  for  a 
scenario  author  to  be  productive  and  reduces 
the  requirement  to  perform  authoring  in  the 
same  development  environment  as  the  testbed. 


•  The  tool  must  provide  the  means  to  easily 
correlate  artifacts  of  the  scenario  with  the 
input  parameters  of  any  applicable  evaluations 
which  require  such  inputs.  For  example,  after 
designating  control  points  along  a  route  in  the 
2D  and  3D  view  of  this  tool,  the  author  should 
be  able  to  easily  cross-reference  a  “Follow 
route”  evaluation  with  these  control  points 
with  a  quick  sequence  of  mouseclicks. 

•  One  of  the  upfront  requirements  for  such  a 
toolset  is  the  development  of  a  library  of 
evaluations  for  the  training  domain, 
representing  a  superset  of  evaluations  across 
anticipated  scenarios.  This  initial  overhead 
will  typically  payoff  when  the  requirement  is 
to  develop  a  plurality  of  scenarios. 

•  When  separate  components  of  a  training 
testbed  run  on  different  terrain  databases,  this 
can  result  in  problems  if  they’re  not  carefully 
correlated.  This  problem  can  be  especially 
pronounced  in  digital  urban  terrain,  where 
there  is  a  significant  difference  between  being 
inside  a  building  versus  outside. 

•  The  representational  format  used  to  capture 
scenario  authoring  data  is  another  technical 
challenge  meriting  future  attention.  For 
example,  the  representational  format  for  OTB 
scenarios  does  not  contain  any  means  for 
capturing  ITS-related  data  such  as  which 
evaluations  apply  with  the  scenario,  and  with 
what  parameters.  Ideally  a  unified, 
generalized  format  would  be  developed  which 
could  accommodate  traditional  forms  of 
scenario  data  in  tandem  with  data  supporting 
structured  training. 

USER  FEEDBACK 

A  key  goal  for  the  research  effort  was  to  collect 
feedback  from  human  test  Participants  using  the  VW 
testbed  with  the  integrated  Intelligent  Structured 
Trainer.  An  experiment  was  conducted  with  a  specific 
scenario  and  set  of  learning  objectives,  and  used  as  a 
basis  on  which  to  elicit  feedback  on  usability, 
suitability,  and  effectiveness. 

Experimental  Design 

The  scenario  created  for  this  experiment  is  based  on 
relevant  activities  encountered  in  the  Contemporary 
Operating  Environment  (COE).  Standards  for  the  tasks 
were  derived  from  the  appropriate  doctrinal  documents 
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and  provided  as  inputs  for  the  implementation  of  the 
Intelligent  Structured  Training  architecture.  The 
scenario  was  then  executed  by  Soldiers  to  determine  if 
the  tasks  trained  accurately  represented  the  proper 
concepts  and  skills ,  elicit  suggestions  for  additional 
tasks  that  could  be  included,  and  collect  comments 
concerning  operating  in  the  virtual  environment. 

The  training  tasks  are  primarily  individual  tasks  which 
the  Soldier  must  accomplish  for  the  collective  task  to 
be  successful.  The  dismounted  Intelligent  Structured 
Training  system  evaluates  three  primary  individual 
tasks.  Soldiers  must  be  able  to: 

•  Move  as  a  member  of  a  fire  team 

•  Send  reports  as  appropriate 

•  Perform  movement  techniques  during  the 
collective  task  of  entering  and  clearing  rooms  . 

Each  of  these  tasks  involves  subtasks  to  be  executed  at 
different  times  and  locations  in  the  virtual  scenario. 
For  the  collective  task,  the  trainer  prompts  the  Soldier 
on  his  individual  actions  within  the  task.  The  standards 
for  the  task  include  the  evaluated  individual  observing 
the  proper  sector  of  fire  during  movement,  position  in 
the  formation,  dispersion  in  the  formation,  and 
maintaining  contact  with  the  fire  team  leader.  All  of 
these  sub-tasks  are  critical  components  to  successful 
completion  of  the  larger  task.  Figure  8  shows  a  run¬ 
time  example  of  feedback  displayed  for  observing  the 
proper  sector  of  fire  as  part  of  Move  as  a  Member  of 
the  Fire  team. 


Figure  8.  Trainer  Feedback  for  Covering  the 
Proper  Sector  of  Fire 

Figure  9  shows  example  feedback  for  sending  an 
observation  report  at  the  proper  time,  determined  from 
the  fact  that  the  enemy  vehicle  is  within  the  field  of 


view,  and  therefore  the  corresponding  report  should 
have  been  sent. 


_ Viitual  Wairioi  I-  llP.IjSl 


Figure  9.  Trainer  Feedback  for  Sending  an 
Observation  Report 


For  the  experiments  conducted,  all  test  Participants 
held  an  Infantry  Military  Occupational  Specialty,  with 
previous  training  in  the  tasks  involved  in  the  scenario . 
Nearly  half  had  combat  experience  performing  the 
tasks  involved  in  the  scenario.  Test  Participants 
received  a  block  of  instruction  on  the  use  of  the 
dismounted  Infantry  system.  They  were  given  time  to 
wear  the  system  and  acclimatize  themselves  to  the 
display,  weapons  mounted  controls,  and  vest  mounted 
controls.  The  Participants  were  also  allowed  to  move 
about  the  virtual  world  to  further  increase  their  comfort 
level.  Immediately  following  the  preparatory  training 
period,  the  Participants  were  given  a  standard  mission 
brief  using  the  mission  planning  tool  with  the  software. 
After  the  mission  brief.  Participants  executed  in  the 
virtual  world  for  30  minutes.  Each  subject  completed 
as  many  iterations  of  the  scenario  as  possible  within  the 
30  minutes.  Upon  completion  of  the  iterations,  the 
Participants  then  answered  a  required  survey. 

The  user  survey  is  organized  into  five  sections.  Section 
one  contained  general  questions  covering  rank,  service, 
combat  experience,  and  familiarity  of  the  tasks 
conducted  in  the  scenario.  Section  two  consisted  of 
questions  pertaining  to  their  assessment  of  the  training 
system.  The  Participants  were  asked  to  indicate  their 
perceptions  of  the  effectiveness  of  the  trainer  as  a  tool 
for  accomplishing  training  objectives.  Section  three 
contained  questions  pertaining  to  their  evaluation  of  the 
testbed  and  whether  or  not  it  allowed  them  to 
accomplish  the  required  tasks  virtually.  Section  three 
also  included  questions  concerning  realism.  The 
Participants  were  asked  a  variety  of  questions 
concerning  training  value,  weapons  effects,  and  the 
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ability  of  the  directive  feedback  to  focus  the  attention 
of  the  subject.  Section  four  consisted  of  the  subject 
ranking  a  list  of  potential  avenues  for  development  in 
building  additional  training  tasks  and  modifying  the 
design  of  the  system.  Section  five  of  the  survey 
contained  a  set  of  questions  concerning  the  individual’s 
response  to  operating  in  the  virtual  world. 

Summary  of  User  Feedback 

The  intent  of  the  assessment  of  the  structured  training 
messages  was  to  determine  if  the  translation  of  the 
standard  for  the  task  from  the  appropriate  doctrinal 
material  to  the  software  implementation  was  a  realistic 
interpretation  of  the  published  standard.  The  data 
gathered  from  the  Participants  provides  their 
assessment  of  the  instructional  aid  to  evaluate  the 
performance  of  the  task.  The  tasks  or  subtasks  that 
were  rated  the  most  favorably  were  the  subtasks  of 
Enter  and  Clear  a  Room,  specifically  Maintaining 
Sector  of  Fire  in  the  Room  and  Maintaining  Muzzle 
Awareness.  The  subtasks  were  rated  much  higher  than 
the  parent  task  of  Enter  and  Clear  a  Room.  This 
suggests  that  it  may  be  generally  easier  to  construct 
effective  training  with  lower  level  isolated  tasks  in  a 
virtual  platform,  as  compared  to  parent-level  inclusive 
tasks. 

A  similar  contrast  was  observed  when  comparing 
responses  regarding  the  parent  task  Move  as  a  Member 
of  a  Fire  Team  with  those  for  the  subtask  Maintain 
Sector  of  Fire  During  Movement .  A  possible  source  of 
this  rating  is  the  use  of  the  C2MINCS  system  at  the 
start  of  the  mission.  The  Participants  were  instructed 
to  report  to  the  higher  unit  headquarters  when  the 
mission  began.  The  act  of  sending  the  report  often 
caused  Participants  to  lag  behind  the  unit  in  the  virtual 
exercise.  This  therefore  caused  difficulty  in  their 
conduct  of  the  Move  as  a  Member  of  a  Fire  team  task. 

The  lowest  rated  tasks  were  Enter  and  Clear  a  Room 
and  Move  as  a  Member  of  a  Fire  Team.  All  of  the 
tasks  with  lowest  ratings  were  tasks  that  required  the 
subject  to  operate  in  conjunction  with  SAF  entities  to 
meet  the  required  standard.  The  SAF  forces  can  be 
difficult  to  work  with,  and  the  majority  of  the  Soldiers 
voiced  displeasure  with  trying  to  accomplish  their 
mission  with  the  SAF  entities.  Representative 
comments  about  the  difficulty  of  working  with  the  SAF 
follow: 

•  “The  SAF  move  too  fast  in  the  scenario.  I 
have  little  time  to  report  before  I  get  left 
behind.” 

•  “There  is  no  way  for  me  to  communicate  with 
the  SAF” 


•  “The  SAF  behavior  is  erratic,  they  are  not 
consistent  fromiteration  to  iteration” 

•  “The  SAF  do  not  follow  a  logical  movement 
pattern  to  the  target  building.” 

This  indicates  a  clear  shortcoming  either  in  the 
implementation  of  SAF  behaviors  for  the  scenario,  or 
in  the  raw  capabilities  of  the  SAF  given  the  constraints 
of  the  OTB  simulation.  A  common  challenge  with 
virtual  dismounted  training  is  that  existing  SAF 
capabilities  may  be  insufficient  to  support  the 
operational  complexity  required  in  a  training  exercise. 
The  prominence  of  user  feedback  on  topics  rooted  in 
SAF-related  performance  also  highlights  the  manner  in 
which  such  issues  can  interfere  with  the  primary 
training  objectives . 

Test  Participants  also  made  several  comments  dealing 
with  exposure  as  a  training  focus.  In  an  urban  fight,  it 
is  extremely  easy  for  a  Soldier  to  become  exposed  to 
the  enemy.  In  the  process  of  clearing  a  room,  Soldieis 
become  fixated  on  the  threat  and  often  disregard 
windows  and  doors  to  adjacent  rooms.  Shot  placement 
was  the  second  highest  ranked  item  for  future 
embedded  training.  This  would  allow  Soldiers  to 
receive  feedback  instantly  on  their  shot  placement.  It 
should  be  noted  that  this  strong  desire  for  training  on 
shot  placement  may  be  an  aberration  due  to  the  lack  of 
feedback  from  the  SAF  enemy  entities  in  the 
implementation  used  in  these  experiments.  They  were 
able  to  absorb  at  least  10  rounds  to  the  body  before 
becoming  neutralized.  The  only  way  to  achieve  instant 
incapacitation  with  the  SAF  enemy  entities  was  to 
score  a  head  shot. 

During  the  scenario  development  stage,  the  initial  force 
structure  for  the  mission  followed  the  doctrinal 
template  required  for  such  a  mission.  Much  of  the 
force  structure  had  to  be  chopped  due  to  erratic 
behavior  of  the  SAF  entities  in  OTB.  To  achieve  a 
minimally  smooth  level  of  interaction  between  SAF 
entities  and  live  players,  some  realism  was 
compromised.  Most  of  the  Participants  were  wary  of 
using  the  technology  of  the  system  to  accomplish  their 
training  goals  versus  using  the  tried  and  true  live 
training  methods.  Most  stated  that  the  technology 
needed  to  grow,  and  they  would  like  to  have  interacted 
with  live  players  in  their  team.  Weapons  effects  are 
also  an  area  that  needed  improvement.  The  sounds 
from  rifles  and  carbines  lacked  a  realistic  sound,  and  as 
stated  earlier  SAF  entities  (dismounted  Infantry)  could 
absorb  an  inordinate  number  of  hits  with  little  effect. 

With  regard  to  the  usability  of  he  player  unit,  he 
majority  of  the  system  was  rated  favorably.  The  lowest 
rated  functions  were  the  ability  to  move  within 
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buildings,  the  location  of  pop-up  messages,  the  use  of 
joystick  and  mouse  simultaneously,  and  the  use  of  the 
C2MINCS  for  reporting  tactical  messages.  Moving 
within  buildings  was  primarily  a  fidelity  issue.  Certain 
buildings  within  the  database  had  glitches  that  made  it 
almost  impossible  to  identify  corridors  for  movement. 
This  problem  generally  presented  itself  in  stairwells  or 
narrow  hallways. 

The  use  of  the  joystick  and  mouse  simultaneously 
could  be  a  workload  issue  for  the  participants.  It  also 
required  the  Soldier  to  violate  sound  tactical  judgment. 
In  order  to  use  the  mouse,  the  Soldier  must  remove  his 
firing  hand  from  the  weapon  due  to  the  non-firing  hand 
controlling  locomotion.  This  is  unnatural  activity  and 
generally  caused  difficulty  for  the  participants. 
Difficulty  with  use  of  the  C2MINCS  could  also  be  a 
workload  issue.  This  was  identified  in  the 
development  stage  of  this  program,  and  several 
modifications  were  made  to  make  sending  reports 
easier.  The  issue  is  that  once  the  C2MINCS  system  is 
employed,  the  focus  is  on  sending  the  required  report, 
and  not  moving  with  the  unit.  By  the  time  the  Soldier 
has  sent  the  report  he  is  already  receiving  messages 
admonishing  him  for  failing  to  meet  standards  for  the 
movement  tasks. 

The  Participants  were  given  a  list  of  five  items  for 
possible  modification  in  future  updated  versions  of  the 
testbed,  to  facilitate  usability  for  training.  These  items 
were  locomotion,  head  mounted  display,  joystick, 
mouse,  and  the  format  of  digital  reports.  The 
Participants  were  asked  to  rank  these  items  numerically 
in  the  order  of  their  need  to  be  modified.  Locomotion 
was  the  top  ranked  choice  for  needed  modification. 
This  modification  would  be  the  most  difficult  issue  to 
implement,  while  still  meeting  the  requirements  of  an 
embedded  training  system.  Any  device  used  for 
locomotion  must  be  small  and  require  minimal 
accessories  to  use.  The  Army  has  a  vision  for 
embedded  devices  to  be  used  as  mission  planning  and 
rehearsal  tools  for  combat  operations.  If  the  devices 
associated  with  the  dismounted  systems  interfere  with  a 
unit’s  ability  to  carry  the  required  tools  for  combat, 
those  devices  will  not  be  deployed  with  the  unit. 


CONCLUSIONS  AND  THE  WAY  FORWARD 

From  a  technical  perspective,  a  primary  finding  from 
the  development  and  integration  effort  is  a  refined 
insight  into  the  nature  of  the  data  needed  to  imp  lenient 
automated  evaluation  mechanisms  for  a  cross-section 
of  dismounted  operational  tasks.  Likewise,  the 
development  of  the  test  scenarios  and  initial  scenario 
authoring  tool  provided  a  view  of  the  key  requirements 


for  effective  training  scenarios.  This  helps  not  only  for 
designing  tools  to  streamline  the  authoring  process,  but 
also  for  noting  the  relationship  between  the  task 
content  of  a  training  scenario  and  the  functional  ability 
of  available  software  and  hardware  to  exercise  tasks 
effectively. 

In  addition  to  the  technical  findings  from  the 
integration  and  development  effort,  the  most  important 
finding  from  the  user  feedback  was  the  degree  to 
which  the  usability  and  suitability  of  the  training 
environment  are  prerequisites  for  effectively  executing 
training  functionality.  Key  performance  factors 
include  the  SAF  ability  to  interact  with  the  trainee  in 
the  performance  of  operational  tasks,  and  the  degree  to 
which  tasks  can  be  performed  within  the  virtual 
environment  where  they  are  afforded  the  opportunity  to 
properly  demonstrate  proficiency. 

For  the  benefit  of  future  research,  it’s  noteworthy  that 
this  study  was  limited  primarily  in  the  number  of 
qualified  personnel  to  undergo  testing.  The  typical 
operational  tempo  of  units  makes  it  difficult  for  a  unit 
to  conduct  a  study  such  as  this  during  training  time, 
which  is  always  limited.  This  led  to  the  significant  role 
of  SAF  entities  in  the  scenario .  The  SAF  entities  could 
not  communicate  with  the  individual  nor  reliably  vary 
their  course  of  action  based  on  enemy  influences.  The 
main  lesson  learned  with  testing  and  experimentation 
would  be  to  include  three  additional  Participants  to 
round  out  the  fire  team  in  the  virtual  environment. 
This  would  have  provided  a  better  test  environment 
and  allowed  more  communication  among  team 
members.  By  comparison,  Michael  Woodman,  in  his 
dissertation  Cognitive  Training  Transfer  using  a 
Personal  Computer-Based  Game:  A  Close  Quarters 
Battle  Case  Study  (Woodman,  2006),  achieved  more 
effective  training  results  when  he  organized  his 
Participants  to  interact  with  three  experts  in  the 
experiment.  The  only  change  from  each  experiment 
run  was  the  subject. 

Additionally,  future  work  that  focuses  on  building 
highly  correlated  terrain  databases  hr  virtual  urban 
operations  will  have  a  measurably  positive  impact  on 
the  feasibility  of  virtual  dismounted  training.  The 
challenge  of  supporting  training-related  data  with 
scenario  data  will  also  be  increasingly  met  by  the 
development  of  more  flexible  representational  formats. 
Specifically,  current  development  work  with  more 
modern  SAF,  such  as  the  OneSAF  Objective  System 
(OOS)  is  focusing  on  more  generalized  forms  of 
scenario  data  and  formats  that  can  accommodate 
scenario  data  needed  by  a  wider  range  of  federated 
components. 
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Intelligent  structured  training  for  this  exercise  was 
generally  specific  to  the  scenario ,  requiring  Participants 
to  operate  in  a  specific  area.  Although  methods  were 
explored  from  a  design  perspective  for  generalizing 
evaluation  mechanisms  for  application  both  in  varying 
situations  within  a  scenario  and  across  scenarios,  this 
level  of  abstraction  remains  a  goal  for  future  research. 
Future  applications  could  include  other  dismounted 
virtual  simulator  concepts  like  workstations  and 
CAVEs.  The  concept  for  future  versions  of  the 
structured  trainer  will  allow  the  trainer  to  select  the 
area  of  operation,  select  the  tasks  to  be  evaluated,  and 
then  select  the  personnel  to  be  evaluated.  This  process 
needs  to  be  simple  and  require  little  additional  training 
if  the  capability  is  going  to  be  available  at  the  unit 
level.  Further  development  on  the  scenario  authoring 
tools  and  associated  methods  will  also  increase  the 
value  of  this  program,  to  support  training  more 
operational  tasks  in  more  situations. 
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